Systems and methods to select an action using a handheld device by perceptual association

ABSTRACT

Systems and methods are described in which a “one-of-N” selection may be produced based on one or more associations made by a human between a selected object from a group of “N” objects, and visual or acoustic cues produced by a handheld device. Objects may be comprised of real or virtual (i.e., displayed) objects in the environment of the handheld device user. Visual and/or acoustic cues may be generated by the handheld device under the control of the device user. Upon determining a perceived association between a cue and a selected object, the user may indicate a selection using one or more handheld device sensing elements such as a pushbutton, touch sensor, or inertial force measurement. Systems and methods may provide simple and intuitive methods to produce selected actions on the handheld device or on a connected device.

RELATED APPLICATION DATA

The present application claims benefit of co-pending provisionalapplication Ser. No. 63/393,761, filed Jul. 29, 2022, the entiredisclosure of which is expressly incorporated by reference herein.

TECHNICAL FIELD

The present application relates generally to systems and methods for anindividual to perform machine-based interactions using a handheldelectronic device. Although the handheld device may be used by anyone,it may be particularly well-suited for use by a young child, utilizingsimple interactive signalling that lacks requirements for precisionmanual dexterity and/or understanding complex interactive sequences.Systems and methods herein employ techniques within the fields ofcomputer programming, electronic design, firmware design, inertialmeasurement units (IMUs), accelerometers, magnetometers, ergonometricconstruction, device controls, human motor control and human-machineinteractions. Systems and methods may provide a user, especially a youngchild, with an intuitive machine interface to rapidly and/orinstinctively interact within an environment composed of real andvirtual objects.

BACKGROUND

In recent years, the world has become increasingly reliant on portableelectronic devices that have become more powerful, sophisticated anduseful to a wide range of users. However, although children may rapidlyembrace using some aspects of electronics designed for more experiencedusers, young children may benefit from having access to interactiveelectronic devices that are small, light-weight, colorful, playful,informative, ergonomically designed for a child (including beingchild-safe), and easy to use. The systems and methods disclosed hereinmake use of recent advances in the fields of portable displays, soundgeneration on mobile devices using miniature coil speakers (orpiezoelectric speakers and/or haptic units), inertial measurement units(sometimes also referred to as inertial motion units) andtelecommunications.

Two-dimensional visual displays may be composed of any number ofmonochromatic or multi-colored, addressable light-sources or pixels.Displays may range from a single light source (e.g., illuminating anorb, transmitted via a waveguide), to those that are capable ofdisplaying a single number (e.g., seven-segment display) or alphanumericcharacter (e.g., a five-pixel by eight-pixel array), to high-resolutionscreens with tens of millions of pixels. Regardless of scale, displaysare typically implemented as: 1) a two-dimensional array of lightsources (most frequently some form of light-emitting diodes (LEDs)including organic LEDs (OLEDs), or 2) two plates of polarized glass thatsandwich liquid crystal material (i.e., forming a liquid crystaldisplay, LCD) that responds to an electric current to allow differentwavelengths of light from one or more illumination sources (i.e., abacklight) to pass.

Within handheld and/or mobile devices, the vibrations of a miniaturespeaker are generally produced using traditional (i.e., associated withlarger speakers) electromagnetic moving coils or piezoelectric(sometimes referred to as “buzzer”) designs. Vibrations (e.g.,particularly those associated with a visual or acoustic alert) may alsobe generated by a haptic unit (also known as kinesthetic communication).Haptic units generally employ an eccentric (i.e., unbalanced) rotatingmass or piezoelectric actuator to produce vibrations (particularly atthe low end of the audio spectrum) that can be felt.

Inertial measurement unit (IMU), accelerometer and/or magnetometertracking may incorporate any or all combinations of: 1) linearaccelerometers measuring forces generated during movement (i.e.,governed by Newton's second law of motion) in up to three axes ordimensions, 2) gyroscope-based sensing of rotational rates or velocitiesin up to three rotational axes, 3) magnetometers measuring magneticfield (i.e., magnetic dipole moment) including fields generated by theearth, and/or 4) the gravitational pull of the earth (includinggravitational orientation) by measuring forces on an internal mass. Theaccuracy of IMUs, accelerometers and magnetometers varies widely,depending on size, operating range, compensating hardware that may beused for correction of measurements (affecting cost), environmentalfactors including thermal gradients, the availability of individualdevice calibrations, and times required to perform measurements(including integration times for some types of measurements).

Advances in both electronics (i.e., hardware), standardizedcommunications protocols and allocation of dedicated frequencies withinthe electromagnetic spectrum have led to the development of a wide arrayof portable devices with abilities to wirelessly communicate with other,nearby devices as well as large-scale communications systems includingthe World Wide Web and the metaverse. Considerations for which protocols(or combinations of available protocols) to employ within such portabledevices include power consumption, communication range (e.g., from a fewcentimeters to hundreds of meters and beyond), and available bandwidth.

Currently, Wi-Fi (e.g., based on the IEEE 802.11 family of standards)and Bluetooth (managed by the Bluetooth Special Interest Group) are usedwithin many portable devices. Less common and/or older communicationsprotocols within portable devices in household settings include Zigbee,Zwave, IR (infrared), and cellular- or mobile phone-based networks. Ingeneral (i.e., with many exceptions, particularly considering newerstandards), compared with Bluetooth, Wi-Fi offers a greater range,greater bandwidth and a more direct pathway to the internet. On theother hand, Bluetooth, including Bluetooth Low Energy (BLE), offerslower power, a shorter operational range (that may be advantageous insome applications), and less complex circuitry to supportcommunications.

Advances in miniaturization, reduced power consumption and increasedsophistication of electronics, including those applied to displays,IMUs, micro electro-mechanical systems (MEMS) and telecommunicationshave revolutionized the mobile device industry. Such portable deviceshave become increasingly sophisticated, allowing users to concurrentlycommunicate, interact, geolocate, monitor exercise, track health, bewarned of hazards, capture videos, perform financial transactions, andso on. Systems and methods that facilitate simple and intuitiveinteractions with a handheld device, particularly for use by children,may be useful.

SUMMARY

In view of the foregoing, systems and methods are provided herein thatdescribe a light-weight, simple-to-use and intuitive handheld devicethat may be particularly well-suited for machine-based interactions by ayoung child. Although the device may, in part, be accepted by a child asa toy, the computational flexibility embedded within the device mayallow the device to be used as a means for play, embodied learning,emotional support, cognitive development, communications, expressingcreativity, developing mindfulness, and enhancing imagination.Additionally, a portable, light-weight, “fun” handheld device maymotivate physical movement by a child (and adults) including kineticmotions and kinesthetic activities.

According to one aspect, devices, systems, and methods are provided fora handheld device user to indicate perceived (i.e., by the user)associations or cognitive correlations between visual, acoustic and/ortiming cues presented via the handheld device and real or virtualobjects in the environment of the device user (i.e., perceived objectsthat are viewable, audible or sensed by any of the other human senses).Visual cues may be presented on one or more displays or other lightsources (e.g., LEDs) on the handheld device. Similarly, acoustic cuesmay be produced by a speaker or other acoustic source (e.g.,piezoelectric buzzer) on the handheld device. Timing cues may beproduced by device display(s), speaker(s) and/or haptic stimulation.Indications of associations perceived by a device user may utilize oneor more sensing elements of the handheld device. For example, anassociation may be indicated by operating a pushbutton, contact sensor,IMU (sensing movement, acceleration and/or orientation in a predefineddirection, or of sufficient magnitude in any direction), camera or othervisual sensor, microphone, biological sensor, or other input (i.e., withrespect to the handheld device) component.

Presentation of visual, acoustic and/or timing cues to the device useron the handheld device may be controlled by the user via movementsand/or orientations (i.e., in three-dimensional space) of the handhelddevice. As an example involving the use of visual cues, the sensedorientation of the handheld device relative to the gravitational pull ofthe earth (i.e., measured using an embedded IMU) may be used to controldisplayed colors (i.e., from a spectrum of colors) projected by the oneor more handheld device displays. By moving the handheld device todifferent orientations (e.g., relative to the gravitational pull of theearth), the user may control generated colors and subsequently select(e.g., via a pushbutton) a color perceived as best matching a color of aselected object.

As a further example that makes use of auditory cues, a speaker withinthe handheld device may pronounce sounds associated with letters (e.g.,“A”, “B”, “C”, etc.) as the orientation of the handheld device ischanged (i.e., analogous to controlling the hand of a clock, or a volumecontrol knob). The sensed orientation of the handheld device relative tothe magnetic pull of the earth (i.e., using an embedded magnetometer)may be used to control sound generation (i.e., from a library of sounds)broadcast by a handheld device speaker. Upon hearing an associated soundthat best matches the selected object (e.g., an object name that startswith the selected letter), a user may indicate (e.g., via a pushbutton)the selected object.

As another example that makes use of associations based on timing and/orsynchrony of presentations; visual, audible and/or haptic cues may bepresented on the handheld device at various rates (e.g., adjustingintervals between presenting or changing cues) that are dependent onmovements of the handheld device (i.e., sensed based on IMU datastreams). When a user determines that the timing of cues presented bythe handheld device approximately matches and/or is in synchrony withdynamic processes of a perceived object, then an indication may be madeby the user of a perceived association with the selected object. As anexample, if the wheels of virtual buses go “round and round”, aparticular bus may be chosen based on matching the timing and/orsynchrony of wheel rotation with handheld device presentations.Selections based on the timing of movements or changes of perceivedobjects are particularly well suited when motion or changes arerepetitive and/or accompanied by music or singing (e.g., providing atiming reference based on melody or beat).

Associations based on visual cues generated by the handheld device mayentail a wide range of visual characteristics including color, colorpattern(s), texture pattern(s), orientations of patterns or textures(e.g., relative to an aspect of the shape of the object), selectedobject shape, relative object size, a displayed name of the object, andso on. Similarly, associations based on acoustic cues generated by thehandheld device may entail a range of acoustic qualities including asound typically generated by the selected object, a sound typicallyassociated with the selected object, an intensity (i.e., volume) ofsound normally associated with the selected object, a phonetic soundassociated with a name of the selected object, and so on. Associates mayalso be based upon the synchronization (e.g., in-phase or out-of-phase),sequencing (e.g., of a pattern of cues) or timing (e.g., repetitionfrequency) of audio and/or visual cues. Within further examples,associations may be made based on categories of objects (e.g., similaruse or appearance), a mathematical relationship, a related object thatis typically paired with a selected object (e.g., during use), and soon.

Objects may be real (e.g., stuffed toys, body parts, pots, pans, livebutterflies, road signs) or virtual (e.g., displayed or projected). Theselection process may be well-suited to a scenario of a young childassociating objects shown on pages within a book, magazine, poster orsticker. The process does not necessarily require any form of pointingthe handheld device toward any object or object selection indicator(e.g., menu). Along similar lines, the handheld device processor (or anyother processor) may not require either an awareness of precise objectlocations (e.g., typically detected using a camera) under considerationby the device user or, for example, where any focus of attention mightbe on the part of the device user (e.g., via eye tracking and/ormonitoring the perceptibility of particular sounds). The overall processmay only require that a device user generates an indication of aperceived (i.e., cognitive) association or cognitive correlation betweena visual, haptic and/or auditory cue or attribute produced by thehandheld device and a selected object (i.e., from a group of objects) inthe environment of the user.

Different designs and/or manufacturers of IMUs may optionally includeone or more magnetometers within core IMU modules. Modern, chip-basedmeasurements of force including those produced by gyroscopes,accelerometers, an internal mass and/or magnetometers typically employMEMS techniques; however, in some cases, magnetometer-based measurementsin particular may be implemented within separate modules (e.g., withtheir own electronic interface) or within subsystem modules (e.g.,electronically slaved) to a primary IMU module. Along similar lines,force measurements in different dimensions (e.g., generally aboutorthogonal axes) may be implemented using separate electronic modules,for example, affixed to device arms where (e.g., compared with mediallocations) forces in one or more directions may be greater. Withindescriptions herein, IMU data refer to any combinations of measuringforces in one or more axes generated by one or more accelerometers,gyroscopes, internal masses (e.g., sensing gravity) and/or magnetometers(i.e., even if implemented within two or more distinct electronicmodules).

In accordance with an example, a method is provided for a human toselect an action using a handheld device related to two or moreperceived objects in proximity to the human, each of the two or moreperceived objects having at least one distinctive visual attribute, themethod comprising: providing a handheld device operated by the humanthat includes a device processor, at least one device displayoperatively coupled to the device processor, at least one inertialmeasurement unit operatively coupled to the device processor, and atleast one selection sensor operatively coupled to the device processor;acquiring, by the device processor from the at least one inertialmeasurement unit, one or more of accelerometer data, gyroscope data,gravitational orientation data and magnetic orientation data; displayinga displayed visual attribute on the at least one device display that isdependent on one or more of the accelerometer data, the gyroscope data,the gravitational orientation data and the magnetic orientation data;acquiring, by the device processor from the at least one selectionsensor, an indication by the human that the displayed visual attributeis perceived by the human to associate with a selected object of the twoor more perceived objects; and performing the action, by one or both ofthe device processor and a remotely connected processor, based at leastin part on one or both of the selected object and an indication time ofthe indication by the human.

In accordance with another example, a method is provided for a human toselect an action using a handheld device related to two or moreperceived objects in proximity to the human, each of the two or moreperceived objects having at least one distinctive acoustic attribute,the method comprising: providing a handheld device operated by the humanthat includes a device processor, at least one speaker operativelycoupled to the device processor, at least one inertial measurement unitoperatively coupled to the device processor, and at least one selectionsensor operatively coupled to the device processor; acquiring, by thedevice processor from the at least one inertial measurement unit, one ormore of accelerometer data, gyroscope data, gravitational orientationdata and magnetic orientation data; broadcasting an acoustic attributeon the at least one speaker that is dependent on one or more of theaccelerometer data, the gyroscope data, the gravitational orientationdata and the magnetic orientation data; acquiring, by the deviceprocessor from the at least one selection sensor, an indication by thehuman that the acoustic attribute is perceived by the human to associatewith a selected object of the two or more perceived objects; andperforming the action on one or both of the device processor and aremotely connected processor, based at least in part on one or both ofthe selected object and an indication time of the indication by thehuman.

In accordance with yet another example, a method is provided for a humanto select an action using a handheld device related to two or moreperceived objects in proximity to the human, each of the two or moreperceived objects having at least one distinctive visual attribute, themethod comprising: providing a handheld device operated by the humanthat includes a device processor, at least one device displayoperatively coupled to the device processor, and at least one inertialmeasurement unit operatively coupled to the device processor; acquiring,by the device processor from the at least one inertial measurement unit,one or more of control accelerometer data, control gyroscope data,control gravitational orientation data and control magnetic orientationdata; displaying a displayed visual attribute on the at least one devicedisplay that is dependent on one or more of the control accelerometerdata, the control gyroscope data, the control gravitational orientationdata and the control magnetic orientation data; acquiring, by the deviceprocessor from the at least one inertial measurement unit, one or moreof selection accelerometer data, selection gyroscope data, selectiongravitational orientation data and selection magnetic orientation data;determining by the device processor from one or more of the selectionaccelerometer data, the selection gyroscope data, the selectiongravitational orientation data and the selection magnetic orientationdata more, one of a tap on the handheld device by the human and agesture movement of the handheld device by the human, indicating thatthe displayed visual attribute is perceived by the human to associatewith a selected object of the two or more perceived objects; andperforming the action on one or both of the device processor and aremotely connected processor, based at least in part on one or both ofthe selected object and an indication time of determining the selectedobject by the human.

In accordance with a further example, a method is provided for a humanto select an action using a handheld device related to two or moreperceived objects in proximity to the human, each of the two or moreperceived objects having at least one distinctive acoustic attribute,the method comprising: providing a handheld device operated by the humanthat includes a device processor, at least one speaker operativelycoupled to the device processor, and at least one inertial measurementunit operatively coupled to the device processor; acquiring, by thedevice processor from the at least one inertial measurement unit, one ormore of accelerometer data, gyroscope data, gravitational orientationdata and magnetic orientation data; broadcasting an acoustic attributeon the at least one speaker that is dependent on one or more of theaccelerometer data, the gyroscope data, the gravitational orientationdata and the magnetic orientation data; acquiring, by the deviceprocessor from the at least one inertial measurement unit, one or moreof selection accelerometer data, selection gyroscope data, selectiongravitational orientation data and selection magnetic orientation data;determining by the device processor from one or more of the selectionaccelerometer data, the selection gyroscope data, the selectiongravitational orientation data and the selection magnetic orientationdata more, one of a tap on the handheld device by the human and agesture movement of the handheld device by the human, indicating thatthe acoustic attribute is perceived by the human to associate with aselected object of the two or more perceived objects; and performing theaction on one or both of the device processor and a remotely connectedprocessor, based at least in part on one or both of the selected objectand an indication time of determining the selected object by the human.

In accordance with another example, a method is provided for a human toselect an action using a handheld device related to two or moreperceived objects in proximity to the human, at least one of the two ormore perceived objects having at least one changing attribute, themethod comprising: providing a handheld device operated by the humanthat includes a device processor, at least one device haptic unitoperatively coupled to the device processor, at least one inertialmeasurement unit operatively coupled to the device processor, and atleast one selection sensor operatively coupled to the device processor;acquiring, by the device processor from the at least one inertialmeasurement unit, one or more of accelerometer data, gyroscope data,gravitational orientation data and magnetic orientation data; producingtactile stimulations of the human by the at least one haptic unit at astimulation rate that is dependent on one or more of the accelerometerdata, the gyroscope data, the gravitational orientation data and themagnetic orientation data; acquiring, by the device processor from theat least one selection sensor, an indication by the human that thestimulation rate is perceived by the human to associate with one or bothof being synchronized with the changing attribute and being at a similarfrequency of the at least one changing attribute of a selected object ofthe two or more perceived objects; and performing the action, by one orboth of the device processor and a remotely connected processor, basedat least in part on one or both of the selected object and an indicationtime of the indication by the human.

In accordance with a further example, a method is provided for a humanto select an action using a handheld device related to two or moreperceived objects in proximity to the human, at least one of the two ormore perceived objects having at least one changing attribute, themethod comprising: providing a handheld device operated by the humanthat includes a device processor, at least one device displayoperatively coupled to the device processor, at least one inertialmeasurement unit operatively coupled to the device processor, and atleast one selection sensor operatively coupled to the device processor;acquiring, by the device processor from the at least one inertialmeasurement unit, one or more of accelerometer data, gyroscope data,gravitational orientation data and magnetic orientation data; displayingvisual attributes on the at least one device display at a display ratethat is dependent on one or more of the accelerometer data, thegyroscope data, the gravitational orientation data and the magneticorientation data; acquiring, by the device processor from the at leastone selection sensor, an indication by the human that the display rateis perceived by the human to associate with one or both of beingsynchronized with the changing attribute and being at a similarfrequency of the at least one changing attribute of a selected object ofthe two or more perceived objects; and performing the action, by one orboth of the device processor and a remotely connected processor, basedat least in part on one or both of the selected object and an indicationtime of the indication by the human.

In accordance with yet a further example, a method is provided for ahuman to select an action using a handheld device related to two or moreperceived objects in proximity to the human, at least one of the two ormore perceived objects having at least one changing attribute, themethod comprising: providing a handheld device operated by the humanthat includes a device processor, at least one speaker operativelycoupled to the device processor, at least one inertial measurement unitoperatively coupled to the device processor, and at least one selectionsensor operatively coupled to the device processor; acquiring, by thedevice processor from the at least one inertial measurement unit, one ormore of accelerometer data, gyroscope data, gravitational orientationdata and magnetic orientation data; broadcasting sounds on the at leastone speaker at an acoustic rate that is dependent on one or more of theaccelerometer data, the gyroscope data, the gravitational orientationdata and the magnetic orientation data; acquiring, by the deviceprocessor from the at least one selection sensor, an indication by thehuman that the acoustic rate is perceived by the human to associate withone or both of being synchronized with the changing attribute and beingat a similar frequency of the at least one changing attribute of aselected object of the two or more perceived objects; and performing theaction on one or both of the device processor and a remotely connectedprocessor, based at least in part on one or both of the selected objectand an indication time of the indication by the human.

Other aspects and features including the need for and use of the presentinvention will become apparent from consideration of the followingdescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding may be derived by referring to theDetailed Description when considered in connection with the followingillustrative figures. In the figures, like-reference numbers refer tolike-elements or acts throughout the figures. Presented examples areillustrated in the accompanying drawings, in which:

FIG. 1 shows two pages within a book, each page containing objects withdistinctive attributes (e.g., the orientation of a zigzag pattern on thebuttons of the four snowmen and different categories of animals) thatallow selection of any individual object by a human via perceivedassociation(s) with cues or attributes generated using a handhelddevice.

FIG. 2 shows a handheld device grasped by the hand of a young child thatincludes three spherical displays and two pushbutton switches used tomake selections (where the thumb is moved slightly compared with normaluse to reveal the switches).

FIG. 3A follows on from the scenario within the left page shown in FIG.1 , in which the orientation of a zigzag pattern on the three displaysof the handheld device (allowing selection of one of the four snowmen)is controlled by the vertical orientation of the handheld device (e.g.,relative to IMU-based measurements of the gravitational pull of theearth).

FIG. 3B illustrates images on three handheld device displays associatedwith different object categories controlled by a horizontal orientationof the device (e.g., relative to magnetometer-based measurements of theearth's magnetic field), allowing selection of a bird (i.e., as ananimal category) from the group of objects shown on the right page inFIG. 1 .

FIG. 4 illustrates the use of rotational orientation (e.g., manipulatedprimarily via a user's wrist) to control sounds generated on a handhelddevice speaker as a method to select an object based on perceivedacoustic association(s).

FIG. 5 is a flowchart outlining exemplary steps to: 1) generate visualcues or attributes on the handheld device based on device orientation,2) indicate a perceived association between a displayed attribute and aselected object in proximity to the device user via a press of apushbutton, and 3) transmit the identity of the selected object (i.e.,snowman with an approximately vertical zigzag pattern on a button) to aremote processor.

FIG. 6 is a flowchart outlining exemplary steps to: 1) generate acousticcues or attributes based on device orientation, 2) select a musicalinstrument based on the broadcasting of a few notes on the handhelddevice speaker, and 3) play tunes on the handheld device that includethe selected musical instrument.

FIG. 7 is a flowchart outlining exemplary steps to: 1) view categoriesof animals on handheld device display(s) based on device orientation, 2)identify a selected animal category based on a tap or gesture using oneor more IMUs within the handheld device to indicate perceivedassociation(s) between a category and an animal in the environment ofthe device user, and 3) play sound(s) typically associated with theselected animal category (i.e., birds).

DETAILED DESCRIPTION

Before the examples are described, it is to be understood that theinvention is not limited to particular examples described herein, assuch may, of course, vary. It is also to be understood that theterminology used herein is for the purpose of describing particularexamples only, and is not intended to be limiting, since the scope ofthe present invention will be limited only by the appended claims.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art. It must be noted that as used herein and in the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise. Thus, for example,reference to “a compound” includes a plurality of such compounds andreference to “the polymer” includes reference to one or more polymersand equivalents thereof known to those skilled in the art, and so forth.

Where a range of values is provided, it is understood that eachintervening value, to the tenth of the unit of the lower limit unlessthe context clearly dictates otherwise, between the upper and lowerlimits of that range is also specifically disclosed. Each smaller rangebetween any stated value or intervening value in a stated range and anyother stated or intervening value in that stated range is encompassedwithin the invention. The upper and lower limits of these smaller rangesmay independently be included or excluded in the range, and each rangewhere either, neither or both limits are included in the smaller rangesis also encompassed within the invention, subject to any specificallyexcluded limit in the stated range. Where the stated range includes oneor both of the limits, ranges excluding either or both of those includedlimits are also included in the invention.

Certain ranges are presented herein with numerical values being precededby the term “about.” The term “about” is used herein to provide literalsupport for the exact number that it precedes, as well as a number thatis near to or approximately the number that the term precedes. Indetermining whether a number is near to or approximately a specificallyrecited number, the near or approximating unrecited number may be anumber which, in the context in which it is presented, provides thesubstantial equivalent of the specifically recited number.

According to one aspect herein, devices, systems, and methods areprovided for a user of a handheld device to specify or indicate (i.e.,“select”) an object (i.e., a “selected object”) from a number (i.e.,“N”, where N is two or more) of real and/or virtual objects that areperceived (i.e., viewable, audible, and/or or sensed by any of the otherhuman senses) by the user, based on one or more associations perceivedby the user between one or more visual, timing, synchronization and/oracoustic cues or attributes presented on the handheld device and theselected object. One result of this process is making a so-called“one-of-N” selection (i.e., decision), where N is the total number ofobjects that are under consideration by the user at any given time.

The one-of-N selection may then be used by one or more processors toinitiate or modulate one or more actions (i.e., “selected actions”) oractivities on the handheld device itself, and/or to transmit theone-of-N selection to a separate electronic device that, in turn, mayinitiate or modulate actions or activities on that device. To initiateor modulate actions on a remote device, the handheld device may includeone or more telecommunications modules (e.g., Wi-Fi, Bluetooth)configured to communicate between the device processor and one or moreremotely connected processors. Additionally, the time the selection wasmade by the handheld device user may be a component of initiating ormodulating actions on handheld and/or remote electronic devices.

Objects may be real (e.g., toys, appliances, pets) or virtual (e.g.,object displayed on a display screen, shown on a mobile device, printedobject within a book, drawn object within a magazine, object projectedby a projection device including a holographic projector, virtualreality headset or augmented reality glasses). The selection process maynot require pointing any device (or finger, eyes or other body part)toward any object or menu. In addition, object locations do not need tobe known by any machinery. One or more handheld device sensing elements(e.g., pushbutton, contact sensor, IMU, camera, microphone) may be usedto indicate an association perceived by the device user. Processes thatuse handheld device sensing elements to indicate a selection aredescribed in more detail, below.

As introduced in the Background and Summary sections above, IMU datastreams may include of one or more of:

-   -   1. up to three channels (i.e., representing three orthogonal        spatial dimensions typically denoted X, Y and Z) of        accelerometer data,    -   2. up to three channels (i.e., representing rotation around 3        axes often denoted pitch, roll and yaw) of gyroscope rotational        velocities,    -   3. up to three channels (i.e., representing orientation in three        orthogonal dimensions) of magnetometer data representing        magnetic forces (e.g., including the magnetic pull of the        earth), and    -   4. up to three channels (i.e., representing orientation in three        orthogonal dimensions) of inertial forces on an internal mass        data that may include the gravitational pull of the earth.

Optionally, each of the IMU data streams may be expressed astime-varying vectors in up to three spatial dimensions using, forexample, Cartesian, polar and/or spherical coordinate systems.

The selection process includes generating and presenting visual and/oracoustic cues or attributes on the handheld device during manipulationsof the device by the user, sensed within one or more IMU data streams.Manipulations may include translational motion, rotation and/ororientation of the device relative to the magnetic and/or gravitationalpull of the earth. IMU-dependent visual cues may be presented on one ormore device displays or light sources (e.g., LEDs). Alternatively, or inaddition, IMU-dependent acoustic cues may be produced by a speaker orother acoustic means (e.g., piezoelectric source) within the handhelddevice.

Conversion of analog IMU data into a digital form, suitable forprocessing, may use analog-to-digital (A/D) conversion techniques,well-known in the art. IMU sample rates may generally be in a range fromabout 100 samples/second to about 10,000 samples/second where (asintroduced in the Background section, above) higher IMU sample ratesinvolve trade-offs involving signal noise, cost, power consumptionand/or circuit complexity. IMU-dependent control of visual and/oracoustic cues may require sample rates in the lower end of the rangejust described whereas, as described further below, determining thepresence of a tap-based selection indication (e.g., using a finger totap the handheld device or tapping the handheld device against a solidsurface) and particularly determining the location of such a tap on thebody of the handheld device generally requires sample rates in the upperend of the range just described.

Control of the presentation of visual and/or acoustic cues by the deviceuser allows the user to manage rates of presenting new cues, how long anindividual cue is presented (or re-presented), and/or the fidelity auser may desire in making an association with particular cues (e.g., howclosely colors, sounds and/or patterns match while making comparisons).Device orientation data may also be used to express movements of thehandheld device relative to previous movements of the device (e.g., whensensing sequential rotation of the device similar to turning atraditional knob), and/or motions relative to other objects in theenvironment of the device user (e.g., a reference object on a viewabledisplay screen).

Associations based on visual cues generated by the handheld device mayentail a range of visual characteristics including color, colorpattern(s), texture pattern(s), orientations of patterns or textures,object shape, object size, display of a name or description of theobject, and so on. Similarly, associations based on audible cuesgenerated by the handheld device may entail a range of acousticqualities including a sound typically generated by the selected object,a sound typically associated with the selected object, phonetic soundassociated with a name of the selected object, enunciating one or moreletters within the name of the object, and so on. Associations may alsobe made based on object categories (e.g., similar operation, function,product and/or appearance), mathematical relations, a related objectthat is typically paired with a selected object (e.g., during use), andso on.

Considering cognitive abilities to make such associations in moredetail, the ability of humans to associate visual cues or attributeswith objects is foundational to survival. The ability to recognizevisual scenes and objects within long term memory generally exhibits amassive storage capacity and (unlike shorter term memory) requireslittle maintenance for retention. Efforts to train machines to associateimages of objects with similar objects and/or object classifications hasled to an appreciation of the power of visual association in humans.Damage to the area of the occipital and temporal lobes within the visualassociation cortex does not hinder an ability for a person to seeobjects, but removes the ability to recognize them as somethingmeaningful.

A wide range of visual properties or attributes enables one to bothdistinguish an individual object as well as to cognitively group orcluster objects into different classifications or categories. Theseattributes include an object's color, combination of colors, texture,patterns (including patterns that themselves repeat), shape and/or size.Most of these attributes may be combined with orientation (e.g.,relative to one or more edges, axes or surfaces of an object; or toreferences in the environment of the device user, such as a floor orwall) including the orientation(s) of a color pattern, object component,texture or shape.

Visual cues may also include one or more words and/or other descriptorsthat may distinguish or point to a selected object (or a distinctivecomponent of a selected object). For example, displaying a “C” may besufficient, based on the first character of words that describe a groupof objects, to isolate a “cat” among the group of objects that beginwith different letters. Along similar lines, displaying the word “red”may be sufficient to identify a selected object among a group ofdifferently colored objects.

Within further examples, perceived associations between a soundgenerated by the handheld device and a specific object in theenvironment of the user may involve a range of cognitive processes. Ingeneral, sound associations mature early during childhood development.Within a world of (real and virtual) objects constantly producingsounds, distinctions within sounds produced by specific objects (e.g.,rattle, squeaky toy, cat) are readily learned at a young age. Some ofthe earliest abilities to distinguish specific sounds usually involvethose made by a mother or other support person in the environment of achild.

Perceived associations may be based on sound qualities applied todifferent objects (e.g., sound volume, presence of harmonics, bangingnoises) that may help focus association(s) to a specific object in auser's environment. Thus, for example, even if a person has never seenor heard a real tiger, any loud and/or fearsome sound may evoke anassociation with an image of such a large creature (e.g., compared withimages of other objects). Along similar lines, the pitch of a sound mayprovide additional cues toward object associations where, for example,higher pitches (and lesser volumes) are generally associated withsmaller objects. Such sound quality assessments may, for example, allowa person to associate a particular sound with a never-seen-beforemusical instrument.

Sound associations are not limited to the specific sounds made by anobject. For example, cognitively, there may be acoustic associationsbased on the sounds of words that define or describe an object, orletters or phonemes that make up those words. For example, among theEnglish-language words “ball”, “cat” and “door”, associating a word thatbegins with the sound produced by a “B” or ends with a sound produced bya “T” is sufficient to identify one selection among the threecandidates. An ability to mix together, break apart and change thesounds of words (i.e., phonological awareness) adds to an ability of aperson to associate sounds with objects based on vocabulary andlanguage.

Although sound symbolism (also known as sound-meaningfulness andphonetic symbolism) that suggests associations between sound sequencesand meanings within spoken words is language-specific and consideredcontroversial by some, the notion may allow some clustering of soundassociations based on word pronunciations (e.g., the beginning of thewords glitter, glacier, and glow all suggesting shiny objects).

Associations may not be limited to strict visual or acoustic propertiesproduced by a handheld device. Cognitive and/or emotional associationsmay place different objects within specific or several differentcategories. These categories allow an object to be selected based onassociations with other objects in the same or a similar category. Forexample, although a hand and a glove are two separate objects, they mayreadily be associated with each other. Categories or classifications maybe based on how the object operates, its purpose or function, itsproduct (i.e., if it produces something), objects that are typicallyused together, similar names, one or more aspects of appearance, one ormore aspects of any sound(s) made by the object, structuralsimilarities, plant or animal phyla, and so on.

The game of charades is an example of an activity that takes advantageof making such associations among (real or virtual) objects. Cognitiveand/or emotional association may be based on a number of syllables in aword that describes the object, descriptive words that sound similar, arelated object similar in appearance, mathematical relationships,musical pitches, objects that have similar functional uses, objects thatare typically seen together, sounds that are typically heard together,and so on.

Within further examples, associations may be based on the timing and/orsynchrony of visual, audible and/or haptic cues presented on thehandheld device. Intervals between (i.e., controlling the rate of)presenting or changing cues may be dependent on movements of thehandheld device (e.g., device orientation, rotational velocity of devicemovements). When a user determines that the timing of cues presented bythe handheld device approximately matches and/or is in synchrony with atleast one changing or dynamic (i.e., changing over time) attribute of atarget or selected object producing one or more changing (e.g., visual,acoustic) attributes, then an indication may be made by the user of aperceived association with the selected object. As an example, if adevice user views a number of (real or virtual) persons waving (e.g., inan effort to attract attention) at differing frequencies, then aparticular individual may be selected by matching (as closely asdesired) the frequency of waving with a rate of presenting IMU-dependentvisual, audible and/or haptic cues on the handheld device.

During such associations based on timing or synchrony, there may be“mixing and matching” of the types of changing attributes produced bythe handheld device versus those associated with two or more perceivedobjects from which a selection may be made. As an example, hapticstimuli may be delivered (and felt by a device user) at an IMU-dependentrate on the handheld device while listening to the beat of musicalinstruments. When a match in beat (i.e., rate of particular notes)and/or synchrony is perceived, then a selected instrument may beindicated (via the handheld device selection sensor) by the user.Similarly, a user may be controlling and listening to IMU-dependentacoustic broadcasts by the handheld device while observing motions ofreal or virtual objects in the user's environment to select an objectbased on rate and/or synchrony of the observed motions.

Within descriptions herein, any means that evokes an associationperceived by a device user with a specific object in the environment ofthe user may be effective within methods to specify a selected objectfrom a group of objects based on visual display(s) and/or acousticsound(s) generated by the handheld device. Having a device user (i.e.,not a machine) make such associations may broaden the range ofassociations possible and avoid deploying sophisticated programmingstrategies within the handheld device such as machine-learning. In orderto make a unique selection, at least one visual and/or acoustic cueand/or the timing of such cues may be distinctive for each of the Nobjects under consideration by a user at a given time.

Optionally, within further examples, a processor may be aware of suchdistinctive cues or attributes, enabling presentation of only suchdistinctive cues or attributes on the handheld device. Alternatively, arange (e.g., continuous or large number of selections) or spectrum ofIMU-dependent visual and/or acoustic cues may be available to the deviceuser. In the latter case, a “closest match” between a visual or acousticattribute generated on the handheld device and corresponding attributesof objects may be computed to determine a selected object.

As examples, a full color pallet (e.g., with millions of availablecolors) may be viewable on handheld device light sources or displays.Once a color or group (i.e., combination) of colors is selected, anobject that contains a color or group of colors that is most similar toa selected color or group of colors may be determined (e.g., using themetric, ΔE, as defined by the International Commission on Illumination).Alternatively, only the colors or group of colors used to make up (e.g.,predefined) viewable objects may be presented on device displays.

Along similar lines, the orientation of a pattern may be selected usinga high degree of fidelity (e.g., among at least dozens of perceptivelydifferent orientations), allowing a user to select from a large numberobjects containing patterns with perceptibly different patternorientations (e.g., see FIGS. 1 and 3A). Alternatively, only predefinedorientations used to make up viewable objects may be presented on devicedisplays.

Within further examples, any handheld device components that may bemanipulated by a user and “sensed” by a device processor may be used bythe device user to indicate a selection. The indication of a selectionmay generally be thought of as a simple, binary indication when aperceived (i.e., by the device user) association occurs with a selectedobject, based on one or more visual and/or acoustic cues. User selectionindications may be made using pushbuttons, touch sensors (e.g.,measuring resistance), proximity sensors (e.g., measuring changes inelectromagnetic induction or capacitance), scroll wheels, toggleswitches, or any other sensed mechanism available on a handheld devicethat may be manipulated by a user.

As an aid to a user before making final selections, intermediate stepsmay optionally be included, for example, to eliminate one or morepossible selections or initially narrow selections to one or moresubsets of visual and/or acoustic cues. As illustrated in FIG. 2 , aselection (i.e., a perceived association) may be indicated using a(momentary contact) pushbutton (e.g., at 23 b). If one or moreadditional pushbuttons are available (e.g., at 23 a), such pushbuttonsmay, for example, be used to indicate a desire by the device user toeliminate a cue or attribute (or group of cues) from the selectionprocess (i.e., not to be re-presented) or to remember to return to apresented cue at some later time.

Alternatively, or in addition, the timing of button presses may be usedto indicate different user intents such as selection versus eliminationof a presented cue. For example, using just a single pushbutton, asingle press may signify eliminating a cue (analogous to indicating“no”) from the selection process (i.e., keeping the cue from beingpresented again on the handheld device). Within this scenario, a doublepress (e.g., two sequential presses within a time less than two seconds)may indicate selection of an object (analogous to indicating “yes”).

In further examples, a microphone may be used to make selections (or toeliminate cues from the selection process) based on associations by adevice user. Sounds or words may be recognized by a device processor tomake such determinations. For example, the word “yes” may be recognizedas identifying a selected object by the user. The word “no” may be usedto eliminate a cue from further consideration, and the phrase “don'tknow” may be classified by a device processor to allow consideration ofother visual or acoustic cues before returning to the visual or acousticcue just presented (if needed). Recognized sounds may include words,phrases, utterances, phonetic sounds, musical notes, animal sounds,alert sounds, and so on.

In yet further examples of selection schemes, by processing IMU datastreams, a “tap” or nudge of the handheld device may be identified as aresult of intentionally moving and subsequently causing an object (i.e.,a “striking object”) to hit a location on the surface of a handhelddevice (i.e., “tap location”) targeted by the user. A tap may begenerated by the device user by one of tapping a holding hand digit ontothe handheld device, tapping an opposing hand onto the handheld device,tapping an opposing hand digit onto the handheld device, tapping a humanbody part onto the handheld device, and tapping the handheld device ontoa solid object. The striking object may also be a stylus, a stick, apencil or other moveable object.

Within further examples, both the handheld device and another object(e.g., child's toy, another handheld device) may be moved simultaneouslytoward each other. In general, relative movements of the handheld devicecompared with the surface being tapped determine tap characteristics(e.g., peak force, accelerations, computed tap location) versus whichobject is moved (e.g., relative to the ground or other objects in theenvironment of a user). IMU data streams prior to and following the tapmay help to determine whether a striking object was used to tap astationary device, the device was forcefully moved toward anotherobject, or both processes occurred simultaneously.

Differing user indications may be detected by identifying different taplocations and/or (particularly acceleration) forces on the handhelddevice (e.g., tapping a center display in FIG. 2 at 22 b indicating“yes” versus tapping the main body of the handheld device to indicate“no”), tapping different objects (e.g., tapping with a relative “soft”surface finger to indicate “yes” while tapping the handheld deviceagainst a hard surface such as a desk to indicate “no”), and/ordistinguishing different gesture movements (e.g., movements in aclockwise direction to indicate “yes” and movements in acounter-clockwise direction to indicate “no”). By determining gesturedirections and/or tap modes (e.g., forceful versus light tap), differentselection steps (e.g., “yes”, “no”, “don't know”) may be indicated.

Within further examples, a shaking of the handheld device may beidentified based on frequency components of device movement (e.g.,measured using Fourier transformation techniques). Physiologically,shaking may be identified as containing frequencies up to about fourHertz (where higher than this range up to about twelve Hertz isgenerally considered a pathological tremor). Shaking the handheld devicemay be in any direction or alternatively, shake direction and/ororientation may be used to indicate user intent such as “yes”, “no” or“don't know”, as just described.

An advantage of using IMU data streams to identify selection steps iseliminating a need for a separate pushbutton or other sensing component(where an IMU is already used for control of visual and/or acousticprompts). One method to combine both: 1) the control of visual and/oracoustic cues and 2) indicating user selections, involves gesturemovements of the handheld device by the human in one or more directionsthat differ (e.g., not in the same plane or direction) from movements orrotational orientations used to control visual and/or acoustic cues.

Although the sensing of selection indications based on IMU data streamsmay eliminate a need for a pushbutton and/or other sensor, this does notimply a need to eliminate symbols, or contact points on the handhelddevice. Indeed, images of pushbuttons or any other symbols may beapplied to (e.g., painted on) surfaces of the handheld device atdifferent potential tap locations. For example, catering particularly toyoung children, different tap locations may be indicated by brightlycolored circles (or other shapes) on the surface of the handheld device.Optionally, such symbols may be applied using temporary adhesives (i.e.,including as so-called “stickers”) that may be exchanged for differentuser applications or simply for fun as a child ages and/or developsdifferent personal preferences.

Methods to determine the presence of a tap, tap characteristics (e.g.,against a hard versus soft surface), and tap locations on the handhelddevice are further described in co-pending U.S. application Ser. No.17/874,253 filed Jul. 26, 2022, the entire disclosure of which isexpressly incorporated herein by reference.

Within additional examples, the time that a selection is made (i.e.,indicating a perceived association by the device user) may be used as afurther element of controlling actions on the handheld device and/orremote system(s). Selection time may be used to compute one or moreintervals since one or more previous selections, or an interval since anevent within the environment of the device user (e.g., a reaction timesince the selected object was viewable and/or heard by the user). Suchselection timing may be deliberate (i.e., where the device userintentionally times when one or more selections occur) or unforced(i.e., where a stimulus-response or selection interval may be measuredby the device in a manner that is unknown to the user).

Selection times may also be compared to additional reference times inorder to further modulate or control one or more actions. As an example,the time that a selection of a food item may be compared to typical mealtimes to determine whether the food item should be located, placed on agrocery list or ordered for delivery. As a further example, whether toinitiate the playing of a selected game or educational experience (i.e.,as a selected action) may depend on the time the selection was maderelative to a usual bedtime for a young child.

Within yet further examples, aspects of a selected action resulting fromthe indication of an association with a selected object may depend oncontext within the interactive environment. Under some contexts, it maybe advantageous to relax the conditions for performing selected actionsor activities. For example, during educational testing, an action thatincludes a “reward” for the device user may be provided when anassociation is made to a correct answer as well as to an answer that isclose to being correct. A relaxing of the perceived association andselection process may be referred to as “interpretive control” that maybe useful within interactions involving the very young, the elderly, orthose with reduced motor and/or cognitive functions.

Additionally, during activities that, for example, involve youngchildren or individuals who are cognitively challenged, associations mayinvolve significant degree of “guessing”. Under such conditions,interpretive control may include forms of “nudging” (e.g., providingintermediary hints) toward one or more correct or target associations.For example, a young child may not fully understand how to manipulate ahandheld device to attain the display of a specific a color. During suchinteractions, auditory instructions may accompany the color selectionprocess (e.g., broadcasting “hold the wand straight up”), guiding theindividual toward a selection. Similarly, a flashing display and/orrepeating sound (where frequency may be related to how close a cue orattribute is to a particular selection) may be presented as a userapproaches a selection (e.g., a correct answer). On the other hand,incorrect selections or a selection in which there is no apparentassociation (e.g., a color selection that appears not related to anyobject known to a device processor) may be accompanied by an “incorrect”indication (e.g., buzzing sound), or instructional broadcasts and/ordisplays. Further aspects of interpretive control are more fullydescribed in U.S. Pat. No. 11,334,178, filed Aug. 6, 2021, and U.S. Pat.No. 11,409,359 filed Nov. 19, 2021, the entire disclosures of which areexpressly incorporated herein by reference.

Along similar lines, selection processes may optionally include adependence on the “context” of an interactive environment. Based oncontext, a subset of all perceived (real or virtual) objects in theenvironment of the device user may be considered during selectionprocesses. Such object exclusions (along with their cues or attributes)may speed selection processes and/or contribute to more readily makingdesired selections. For example, if a device user is involved in theconstruction of a virtual object, selections made using the handhelddevice may be isolated to the tools and/or building components relatedto the construction project. During one or more construction activities,perceived objects in the environment of the device user that are notassociated with the construction project may be excluded duringselection processes, at least temporarily. Determining context fromaudiovisual content and subsequently generating control based on suchcontexts are more fully described in U.S. Pat. No. 11,366,997, filedApr. 17, 2021, the entire disclosure of which is expressly incorporatedherein by reference.

Additionally, the generation of visual and/or auditory attributes on thehandheld device may take into account accessibility factors by somedevice users. For example, particular colors and/or color patterns maybe avoided within visual cues when devices are used by individuals withdifferent forms of color blindness. Similarly, if an individual has ahearing loss over one or more ranges of audio frequencies, then thosefrequencies may be avoided or boosted in intensity (e.g., depending onthe type of hearing loss) within audio cues.

The handheld device may additionally include one or more of one or morephotodiodes, an optical blood sensor, and an electrical heart sensor,each operatively coupled to the device processor. These additionalcomponents may provide additional options for user select and/ormodulate actions or cues using the handheld device. In addition, abattery, providing power to the electronic components, may allow thehandheld device to operate untethered from any other power source.

The handheld device may optionally include one or more cameras or otherlight sensors (e.g., scanners, bar code readers). Such one or more lightsensors may be used to determine one or more visual attributes ofperceived objects in the environment of the handheld device user duringa selection process. As an example, a camera may determine distinctivecolors of drawn objects within viewable pages of a book. Such colors maythen be used as a basis for selecting from the objects within theviewable pages based on perceived color matching. As a further example,a camera or QR code reader may be used to identify a selection ofobjects displayed on a screen. An individual object may subsequently beselected based on a unique visual and/or acoustic attribute of theidentified objects (e.g., with known visual or acoustic characteristicsor attributes based on the one or more QR codes).

The handheld device may also optionally include one or more microphonesthat, similar to light-based sensors just described, may be used toidentify acoustic cues of objects in the environment of the handhelddevice user. As an example, a microphone may identify repetitive soundsfrom different objects (e.g., different birds chirping). An individualbird may subsequently be selected based on a selecting a rate (e.g.,based on haptic stimulation by the handheld device) that matches orsynchronizes with chirping perceived by the device user.

The physical location of a handheld device as well as its orientation in(three-dimensional) space may be further determined by camera-basedtracking of the handheld controller during at least some times when inuse such as when visible to one or more cameras (i.e., where suchcameras are not a component of the handheld device). Camera-basedmeasures of handheld controller position (e.g., orientation in thefield-of-view of the camera including relative to other objects in thefield-of-view, velocity, acceleration) may be combined with IMU-baseddata streams to provide further initiation and/or modulation ofresultant actions. Systems and methods to determine such camera-basedmeasures are described in U.S. Pat. No. 11,334,178, filed Aug. 6, 2021and U.S. Pat. No. 11,409,359, filed Nov. 19, 2021, the entiredisclosures of which are expressly incorporated herein by reference.

Within additional examples, although not “handheld” in a strict sense,such portable electronic devices may be affixed and/or manipulated byother parts of the human body. A device in which perceived associationsare made based on IMU-dependent visual and/or acoustic presentationsmay, for example, be affixed to an arm, leg, foot or head. Suchpositioning may be used to address accessibility issues for individualswith restricted upper limb and/or hand movement, individuals absent ahand, and/or during situations where a hand may be required for otheractivities.

During some situations such as when N (i.e., the number of perceivedobjects) is large, it may not be practical or physiologically feasibleto distinguish a sufficient number of visual or acoustic attributes toselect from a group of N objects in one step. For example, ifassociations are based on object color(s) within a book contains dozensof pages and where each pages contains dozens of objects, then it maynot be economically feasible to print such a book with up to thousandsof distinct object colors or to have a device user make subtledistinctions among such a large number of color selections (particularlyif including accommodations for color blindness). Even within a singlepage, selecting a single state or province from a map containing fiftyor more distinctly color-coded states or provinces would be difficultfor many device users.

As N becomes large, it may be advantageous to layer the selectionprocess using attributes generated on the handheld device, where suchlayered selection-making may be expressed or visualized in the form of aso-called “decision or selection tree”. For example, if a book containsa large number of pages, then a device user may initially select achapter within the book (e.g., by displaying chapter number orbroadcasting the titles of chapters using a handheld device speaker).The next (decision or selection tree) layer may be to select a pagewithin the selected chapter (again, either by visual or acousticassociation), followed by a selection process for specific objectsdisplayed on a selected page. Selection of a color-coded state orprovince within a map may entail dividing the map into regions (and evenfurther into sub regions) before selecting a specific state or province.

Using such a decision-tree approach, the total number of possibleselections is the (multiplicative) product of the number of possibleselections of each layer. Different layers or stages within thedecision-tree process may use visual attributes, acoustic attributes orboth simultaneously. There is no dependence on association modes (e.g.,visual, acoustic, categories of objects) used within any layer or stageof the process with any other selection stage. In other words, there isfreedom to “mix-and-match” the types of associations made by a deviceuser at any time during such a decision-tree selection process.

Within further examples, it is even possible to generate multiple(visual and/or acoustic) attributes during a single selection step. Forexample, if a user hesitates in producing additional visual or acousticcures, or in making a selection (e.g., for a period greater than about afew seconds), aspects of the attribute may be repeated, changed (e.g.,made brighter, made louder), and/or a different type or representationof an attribute may be displayed or broadcast (e.g., a differentrepresentation of an object category, different name). The repeating ofcues or attributes, including utilizing different forms, may beperformed any number of times, particularly following prolongedhesitation by a user.

FIG. 1 shows two pages within a book or magazine 10, where each of thetwo pages Ila, 11 b contain objects 12 a, 12 b, 12 c, 12 d, 13 a, 13 b,13 c, 13 d with distinctive visual attributes. Perceived associationsbetween visual and/or acoustic attributes and distinctive imagesdisplayed on book pages may be selected using a handheld device (notshown).

On the left page 11 a of the book 10, the visual orientation of a zigzagpattern on the buttons 14 a, 14 b, 15 a, 15 b, 16 a, 16 b, 17 a, 17 b ofthe four snowmen 12 a, 12 b, 12 c, 12 d distinguish the snowmen fromeach other. For example, the zigzag pattern is horizontal (i.e.,relative to the typical orientation while reading the book) within thepair of buttons 16 a, 16 b on the snowman at 12 c on the lower left ofpage at 11 a. The horizontal pattern 16 a, 16 b is distinctive from allother buttons 14 a, 14 b, 15 a, 15 b, 17 a, 17 b.

Within the right page 11 b of the book 10, each of the images 13 a, 13b, 13 c, 13 d depicts a different type of animal (i.e., bird at 13 a,cat a 13 b, bear at 13 c and cow at 13 d). Thus, for example, an imageand/or sound of a bear produced by a handheld device (not shown), wouldbe perceived by most individuals as associated with the cartoon-likeimage at 13 c depicting a bear on the lower left side of the book pageat 11 b.

FIG. 2 shows a handheld device 20 grasped by the right hand 21 of ayoung child. The handheld device 20 includes three spherical displaycomponents 22 a, 22 b, 22 c, two pushbuttons 23 a, 23 b and an embeddedspeaker, e.g., just inside the skin of the device in the region outlinedby a dashed-line ellipse 24. The thumb 25 of the child's hand 21 ismoved slightly compared with normal use to reveal two device pushbuttons23 a, 23 b. Visual cues or attributes that may be associated with aselectable object in the vicinity of the device user may be displayed onany or all of the device displays at 22 a, 22 b, 22 c and/or acousticcues may be broadcast using the device speaker at 24.

FIG. 3A follows on from the scenario depicted within the left page ofthe book in FIG. 1 , in which the orientation of a zigzag pattern onpairs of buttons (14 a, 14 b, 15 a, 15 b, 16 a, 16 b, 17 a, 17 b in FIG.1 ) within each of four snowmen (12 a, 12 b, 12 c, 12 d in FIG. 1 ) is avisual feature that distinguishes individual snowman from each other(and other objects within the pages shown). FIG. 3A shows a handhelddevice 30 being cradled between a thumb 35 a and fingers 35 b, 35 c, 35d, 35 e of a right hand 31 of a device user. The thumb may also be usedto operate a (momentary contact) pushbutton at 33 to indicate selectionsby the device user.

In this selection example, zigzag patterns 36 a, 36 b, 36 c aredisplayed on each of the three spherical displays 32 a, 32 b, 32 c ofthe handheld device 30. In FIG. 3A, the handheld device 30 is shownoriented vertically and may subsequently be rotated 37 a by the deviceuser in a manner analogous to rotating a hand of a clock on a wall(i.e., in a frontal plane) to control rotational orientation of thedisplayed zigzag patterns 36 a, 36 b, 36 c. IMU-based data streams allowdetermination of the direction (i.e., in three-dimensional space) of thegravitational pull of the earth that, in turn, allows handheld deviceorientation to be computed. Optionally during this time, the handhelddevice may additionally produce acoustic cues or attributes via thedevice speaker 34 as the handheld device is rotated 37 a.

Once the device user perceives that the zigzag pattern on the devicedisplays 36 a, 36 b, 36 c match (at least approximately) the zigzagpattern of a pair of the buttons on a selected snowman (i.e., thesnowman at 12C shown on the lower left of the page 11 a in FIG. 1 ), anindication by the device user may be made using a thumb at 35 a to pressa pushbutton at 33 on the handheld device 30. This selection, along witha recorded time that the selection was made, may then be used to controlsubsequent actions or activities with the handheld device 30 and/ortransmitted to other devices (not shown) to control or modulateadditional or alternative actions.

FIG. 3B follows on from the scenario depicted within the left page ofthe book or magazine in FIG. 1 , in which images representing a bird at13 a, cat at 13 b, bear at 13 c and cow at 13 d are displayed. Thehandheld device 30 is cradled between the thumb 35 a and fingers 35 b,35 d, 35 e of the right hand 31 of a device user. By rotating and/ormoving the handheld device in a horizontal plane (i.e., parallel to thesurface of the earth 37 b), different classes of objects may be shown ondevice displays 36 a, 36 b, 36 c. Rotational orientation may bedetermined based on a magnetometer data stream gathered from an IMUembedded within the handheld device 30 where orientation may bedetermined relative to the magnetic pull of the earth. Alternatively, orin addition, translational movements (e.g., in the horizontal plane) maybe sensed by one or more IMU accelerometers to view different objectsand/or categories of objects on the displays 36 a, 36 b, 36 c of thehandheld device 30 as the device is moved about.

In these exemplary cases, a category of objects may be selected usingthe thumb at 35 a of the hand 31 controlling the handheld device 30 topush a pushbutton at 33 when there is a perceived association by thedevice user between the image category displayed on the central displayat 36 b of the device 30 and the selected object category. For example,a press of the pushbutton at 33 as shown in FIG. 3B indicates theselection of a bird or general category of birds (e.g., corresponding tothe upper-left object at 13 a on the right page 11 b of the book 10illustrated in FIG. 1 ).

Upon rotating clockwise (i.e., looking down on the device from theperspective shown in FIG. 3B) and/or moving the device translationallyto the right, a new image appears on the right display at 36 c, theright display image (i.e., an image of a whale) moves to the centerdisplay at 36 b, and the center image (i.e., the image of a bird) movesto the left display at 36 a. Conversely, moving handheld device 30 in acounter clockwise direction and/or moving the handheld devicetranslationally to the left, a new image appears on the left display at36 a, the left display image (i.e., an image of a boat) moves to thecenter display at 36 b, and the center image (i.e., the image of a bird)moves to the right display at 36 a. In this example, different imagesappear on each device display, allowing a device user to “look ahead”(as well as to maintain a sense of relative object positions with avirtual space) to see objects in each direction during the process ofbringing an object category to the central display at 36 b as a part ofthe process to make an object or object category selection.

FIG. 4 illustrates user-controlled broadcasting of different sounds on ahandheld device speaker 44 as component of a process to indicate anacoustic association between sound(s) produced by the handheld deviceand one or more sounds associated with a selected object. As describedin more detail above, associations with object sounds perceived by adevice user may, for example, be based on sound(s) the object produces,sound(s) within words that describe the object, sound(s) produced by acategory of objects that include the selected object, and so on.

Illustrating yet another means for handheld device control, in FIG. 4 ,rotation 47 of the handheld device 40 about its long axis may be used asa method to control the broadcasting of different sounds 46 on one ormore speakers 44 of the handheld device 40 (e.g., controlling one ormore rotational movements often referred to a pitch, roll and/or yaw).Cradled within the hand 41 of the device user, the device 40 may berotated (i.e., altering roll), for example, by manual manipulation(particularly using the thumb 45) or by twisting particularly at thelevel of the wrist (not shown). One or more sounds generated on thehandheld device speaker may be used as a means to select an object basedon perceived acoustic association(s) by a device user. The broadcastingof different sounds 46 may optionally be accompanied by visual cues orattributes on device displays 42 a, 42 b, 42 c.

As components of the handheld device 40 shown in FIG. 4 , two (momentarycontact) pushbutton switches 43 a, 43 b are available to the deviceuser. One switch (e.g., at 43 b) may, for example, be used to indicate aperceived association between a sound being broadcast by the device 46and a selected object. When stepping through different sounds as thehandheld device is rotated 47, the second pushbutton (e.g., at 43 a)may, for example, be used to eliminate the sound(s) of possibleselections. In other words, if the device user wishes to eliminate asound from a repertoire of sounds (and perhaps replay a previouslybroadcast sound) a press of the second pushbutton (e.g., at 43 a) may beused to indicate “no” or “not this one”.

Along similar lines, other sensors (i.e., input components with respectto the handheld device 40) may be used to isolate and/or focus in onsounds perceived to associate with a selected object. For example, amicrophone embedded within the handheld device (not visible in FIG. 4 )may be used to control, select and/or eliminate different sounds (e.g.,via device-based recognition of one or more distinct keywords orphrases). Optionally, when using a microphone, a pushbutton (e.g., at 43b) may be used to alert the device to “listen” (i.e., machine monitoringfor identifiable keywords or sounds). The device may, for example, seeka match in the audio data stream for phrases such as “yes”, “no”, “notthis one”, and “save” (i.e., for re-review at a later time).

FIGS. 5, 6 and 7 are flowcharts outlining exemplary steps to select anaction using a handheld device based on visual and/or acousticassociations made by the device user. From an overall perspective,different activities or actions may be achieved by “mixing and matching”key steps illustrated within these flowcharts. For example, visual(e.g., as in FIGS. 5 and 7 ), acoustic (e.g., as in FIG. 6 ) or bothforms of stimuli may be used to form associations by the device userwith an object in the environment of the user. Sensing by the handhelddevice to determine when the device user indicates a perceivedassociation may be based on the states of pushbutton or contact switches(e.g., as in FIG. 5 ), words or other sounds recognized within datagathered by a handheld device microphone (e.g., as in FIG. 6 ), and/orthe identification of a tap or gesture based on data gathered by one ormore IMUs (e.g., as in FIG. 7 ) within the handheld device. An actionperformed as a result of a selection may be performed by the handhelddevice itself (e.g., as in FIGS. 6 and 7 ) or by a separate, connecteddevice (e.g., as in FIG. 5 ). Actions may involve acoustic (e.g., as inFIGS. 6 and 7 ), visual and/or other forms of interaction such as hapticfeedback, physical control of internet-of-things based objects (lightswitches, thermostat controls), and so on. Most combinations of suchselection steps and resultant actions are possible.

FIG. 5 is a flowchart outlining strategies to indicate a perceivedassociation between a displayed cue or attribute and a selected objectin the environment of a device user via a pushbutton press 53 b, and totransmit the identity of the selected object (and a time the object wasselected) to a remote processor 59 b. Steps in this exemplary processinclude:

-   -   1. at 50 a, grip the handheld device 51 a in a manner that        allows the user to visualize device displays as well as to        permit access to at least one selection pushbutton at 53 a that        may be depressed by a thumb 55 a (or other digit) of the hand of        the device user;    -   2. at 50 b, acquire IMU data (typically in three orthogonal        directions 56), e.g., based on the gravitational and/or magnetic        pull of the earth, to compute the orientation and/or movements        of the handheld device 51 b;    -   3. at 50 c, display visual attributes on device displays 52 a,        52 b, 52 c that are dependent on handheld device 51 b        orientation (e.g., the rotational orientation of a zigzag        pattern) and sense the depressing of a pushbutton by the thumb        of the device user 55 b to select an object based on the        displayed visual attribute(s);    -   4. at 50 d, determine the state of a selection pushbutton 53 b        and time(s) of any changes in state 53 c;    -   5. at 50 e, determine if a selection by the device user was        enacted (via the pushbutton) and, if not, return at 54 to step        50 b;    -   6. at 50 f, register the time a selection by the device user was        made 57 a as well as any intervals 57 b between the current        selection time and previous selection times and/or events in the        environment of the device user;    -   7. at 50 g, from a group of objects in the (real or virtual)        environment of the device user 58 a, identify a selected object        58 b that best matches the visual attribute(s) displayed (at        step at the time the user selection was made;    -   8. at 50 h, transmit from the handheld device 59 a to one or        more remote processors 59 b the user selection and the time the        user selection was made, allowing the remote system(s) to        perform one or more actions based on the user selection and/or        the time the selection was made.

FIG. 6 is a flowchart outlining processes to select a musical instrumentbased on the broadcasting of a few musical notes on the handheld devicespeaker 64 a, and to subsequently play tunes that include the selectedmusical instrument 60 h. Steps in this exemplary process include:

-   -   1. at 60 a, grip the handheld device 61 a in a manner that        allows the user to hear sounds produced by the device speaker 64        a;    -   2. at 60 b, acquire IMU data (typically in three orthogonal        directions 66), e.g., based on the gravitational and/or magnetic        pull of the earth, to compute the orientation and/or movements        of the handheld device 61 b;    -   3. at 60 c, broadcast musical notes (and/or other acoustic        attributes) on the handheld device speaker 64 b that are        dependent on handheld device 61 b IMU-measured orientation;    -   4. at 60 d, using a microphone 63 b, listen to sounds 63 a in        the environment of the handheld device to determine to determine        the presence of an identified selection sound;    -   5. at 60 e, determine if a selection (or keyword) sound has been        identified within microphone data and, if not, return at 65 to        step 60 b;    -   6. at 60 f, register the time a selection by the device user was        made 67 a as well as any intervals 67 b between the current        selection time and previous selection times and/or events in the        environment of the device user;    -   7. at 60 g, from a group of objects (e.g., flute, trumpet,        violin, saxophone, bassoon) in the (real or virtual) environment        of the device user 68 a, identify a selected object 68 b (e.g.,        trumpet) that best matches the acoustic attribute(s) broadcast        (at step 60 c) at the time the user selection was made;    -   8. at 60 h, broadcast 62 b on the one or more speakers 64 c of        the handheld device 61 c, musical tunes that include the        selected instrument 68 b.

FIG. 7 is a flowchart outlining operations to identify a category ofanimals based on visual cues with a selection indicated by a tap orgesture of the handheld device, and to play sound(s) typicallyassociated with the selected animal (i.e., birds). In this example,there is a mixing of visual cues with selected acoustic action(s). Stepsin this exemplary process include:

-   -   1. at 70 a, grip the handheld device 71 a in a manner that        allows the user to visualize device displays as well as to        readily move the device about;    -   2. at 70 b, acquire IMU data (typically in three orthogonal        directions 76 a), e.g., based on the gravitational and/or        magnetic pull of the earth, to compute the orientation and/or        movements of the handheld device 71 b;    -   3. at 70 c, display visual attributes comprising different        categories of objects (e.g., boats, birds, whales) on device        displays 72 a, 72 b, 72 c that are dependent on handheld device        71 b IMU-measured orientation;    -   4. at 70 d, acquire additional IMU data 76 b, particularly from        accelerometers and gyroscopes, to determine the presence of a        tap or gesture;    -   5. at 70 e, determine if a recognizable tap or gesture is        present within IMU data and, if not, return at 75 to step 70 b;    -   6. at 70 f, register the time a selection by the device user was        made 77 a as well as any intervals 77 b between the current        selection time and previous selection times and/or events in the        environment of the device user;    -   7. at 70 g, from a group of objects (e.g., bird, cat, bear, cow)        in the real or virtual environment of the device user 78 a,        identify a selected object 78 b that is perceived by the device        user as being most closely associated with a selected object        category (e.g., bird) displayed (at step 70 c) at the time the        user selection was made;    -   8. at 70 h, broadcast sounds 73 on speaker 74 of the handheld        device 71 c that are typically produced by the selected object.

The foregoing disclosure of the examples has been presented for purposesof illustration and description. It is not intended to be exhaustive orto limit the invention to the precise forms disclosed. Many variationsand modifications of the examples described herein will be apparent toone of ordinary skill in the art in light of the above disclosure. Itwill be appreciated that the various components and features describedwith the particular examples may be added, deleted, and/or substitutedwith the other examples, depending upon the intended use of theexamples.

Further, in describing representative examples, the specification mayhave presented the method and/or process as a particular sequence ofsteps. However, to the extent that the method or process does not relyon the particular order of steps set forth herein, the method or processshould not be limited to the particular sequence of steps described. Asone of ordinary skill in the art would appreciate, other sequences ofsteps may be possible. Therefore, the particular order of the steps setforth in the specification should not be construed as limitations on theclaims.

While the invention is susceptible to various modifications, andalternative forms, specific examples thereof have been shown in thedrawings and are herein described in detail. It should be understoodthat the invention is not to be limited to the particular forms ormethods disclosed, but to the contrary, the invention is to cover allmodifications, equivalents and alternatives falling within the scope ofthe appended claims.

1. A method for a human to select an action using a handheld devicerelated to two or more perceived objects in proximity to the handhelddevice, each of the two or more perceived objects having at least onedistinctive visual attribute, the method comprising: providing ahandheld device operated by the human that includes a device processor,at least one device display operatively coupled to the device processor,at least one inertial measurement unit operatively coupled to the deviceprocessor, and at least one selection sensor operatively coupled to thedevice processor; acquiring, by the device processor from the at leastone inertial measurement unit, one or more of accelerometer data,gyroscope data, gravitational orientation data and magnetic orientationdata while the human is manipulating the handheld device to identifymovement of the handheld device by the human; displaying one of aplurality of displayed visual attributes on the at least one devicedisplay as the handheld device is manipulated by the human that isdependent on one or more of the accelerometer data, the gyroscope data,the gravitational orientation data and the magnetic orientation data;acquiring, by the device processor from the at least one selectionsensor, an indication by the human that a displayed visual attributecurrently displayed on the at least one display is perceived by thehuman to be associated with a selected object of the two or moreperceived objects; and performing the action, by one or both of thedevice processor and a remotely connected processor, based at least inpart on one or both of the selected object and an indication time of theindication by the human.
 2. The method of claim 1, wherein the two ormore perceived objects each comprises one of a printed object within abook or a drawn object within a magazine.
 3. The method of claim 1,wherein the at least one inertial measurement unit includes one or moreof one or more accelerometers, one or more magnetometers, internal masssensors and one or more gyroscopes.
 4. The method of claim 1, whereinone or more of the accelerometer data, the gyroscope data, thegravitational orientation data and the magnetic orientation data eachcomprises measurements in directions orthogonal to each other.
 5. Themethod of claim 1, wherein the displayed visual attribute comprises oneor more of a displayed solid color, a displayed color combination, adisplayed pattern, a displayed pattern orientation, a displayed texture,a displayed texture orientation, a displayed object shape, a displayedobject size, a displayed category of objects, a displayed mathematicalrelationship, a displayed related object typically paired with theselected object, and one or more displayed characters included within aname of the selected object.
 6. The method of claim 1, wherein the oneor more selection sensors each comprises one of a pushbutton, a switch,a touch sensor, a proximity sensor, and a microphone.
 7. The method ofclaim 1, wherein the one or more selection sensors comprise a microphoneand wherein acquiring the indication comprises identifying a recognizedsound made by the human identified within data acquired from themicrophone by the device processor.
 8. The method of claim 7, whereinthe recognized sound comprises one or more of one or more words, one ormore phrases, one or more utterances, one or more phonetic sounds, oneor more musical notes, one or more animal sounds, and one or more alertsounds.
 9. The method of claim 1, wherein the action is further based atleast in part on one or both of an indication interval between theindication time and a previous time of a previous indication by thehuman, and an event interval between the indication time and an eventtime of an event in proximity to the human.
 10. The method of claim 1,wherein an additional action by one or both of the device processor andthe remotely connected processor, based at least in part on one or bothof an additional selected object and an additional indication time of anadditional indication by the human, is performed by: acquiring, by thedevice processor from the at least one inertial measurement unit, one ormore of additional accelerometer data, additional gyroscope data,additional gravitational orientation data and additional magneticorientation data; displaying an additional displayed visual attribute onthe at least one device display that is dependent on one or more of theadditional accelerometer data, the additional gyroscope data, theadditional gravitational orientation data and the additional magneticorientation data; and acquiring by the device processor from the atleast one selection sensor, the additional indication by the human thatthe additional displayed visual attribute is perceived by the human toassociate with the additional selected object of the two or moreperceived objects.
 11. The method of claim 1, wherein the device furthercomprises one or both of a Wi-Fi communications module and a Bluetoothcommunications module configured to communicate between the deviceprocessor and the remotely connected processor.
 12. A method for a humanto select an action using a handheld device related to two or moreperceived objects in proximity to the handheld device human, each of thetwo or more perceived objects having at least one distinctive acousticattribute, the method comprising: providing a handheld device operatedby the human that includes a device processor, at least one speakeroperatively coupled to the device processor, at least one inertialmeasurement unit operatively coupled to the device processor, and atleast one selection sensor operatively coupled to the device processor;acquiring, by the device processor from the at least one inertialmeasurement unit, one or more of accelerometer data, gyroscope data,gravitational orientation data and magnetic orientation data while thehuman is manipulating the handheld device to identify movement of thehandheld device by the human; broadcasting one of a plurality ofacoustic attributes on the at least one speaker as the handheld deviceis manipulated by the human that is dependent on one or more of theaccelerometer data, the gyroscope data, the gravitational orientationdata and the magnetic orientation data; acquiring, by the deviceprocessor from the at least one selection sensor, an indication by thehuman that an acoustic attribute currently broadcast by the at least onespeaker is perceived by the human to be associated with a selectedobject of the two or more perceived objects; and performing the actionon one or both of the device processor and a remotely connectedprocessor, based at least in part on one or both of the selected objectand an indication time of the indication by the human.
 13. The method ofclaim 12, wherein the acoustic attribute comprises one or more of one ormore words, one or more pronunciations of one or more letters within aname of the selected object, one or more elements of a spokenexpression, one or more spoken mathematical relationships, one or morephonetic sounds, one or more object sounds typically produced by theselected object, one or more sound categories, one or more musicalnotes, alert sounds, and one or more animal sounds.
 14. The method ofclaim 12, wherein the one or more selection sensors each comprises oneof a pushbutton, a switch, a touch sensor, a proximity sensor, and amicrophone.
 15. The method of claim 12, wherein the one or moreselection sensors comprise a microphone and wherein acquiring theindication comprises identifying a recognized sound made by the humanidentified within data acquired from the microphone by the deviceprocessor.
 16. The method of claim 15, wherein the recognized soundcomprises one or more of one or more words, one or more phrases, one ormore utterances, one or more phonetic sounds, one or more musical notes,one or more animal sounds, and one or more alert sounds.
 17. A methodfor a human to select an action using a handheld device related to twoor more perceived objects in proximity to the handheld device, each ofthe two or more perceived objects having at least one distinctive visualattribute, the method comprising: providing a handheld device operatedby the human that includes a device processor, at least one devicedisplay operatively coupled to the device processor, and at least oneinertial measurement unit operatively coupled to the device processor;acquiring, by the device processor from the at least one inertialmeasurement unit, one or more of control accelerometer data, controlgyroscope data, control gravitational orientation data and controlmagnetic orientation data while the human is manipulating the handhelddevice to identify movement of the handheld device by the human;displaying one of a plurality of displayed visual attributes on the atleast one device display as the handheld device is manipulated by thehuman that is dependent on one or more of the control accelerometerdata, the control gyroscope data, the control gravitational orientationdata and the control magnetic orientation data; acquiring, by the deviceprocessor from the at least one inertial measurement unit, one or moreof selection accelerometer data, selection gyroscope data, selectiongravitational orientation data and selection magnetic orientation data;determining by the device processor from one or more of the selectionaccelerometer data, the selection gyroscope data, the selectiongravitational orientation data and the selection magnetic orientationdata more, one of a tap on the handheld device by the human and agesture movement of the handheld device by the human, indicating that adisplayed visual attribute currently displayed on the at least onedevice display is perceived by the human to be associated with aselected object of the two or more perceived objects; and performing theaction on one or both of the device processor and a remotely connectedprocessor, based at least in part on one or both of the selected objectand an indication time of determining the selected object by the human.18. The method of claim 17, wherein the gesture movement of the handhelddevice by the human comprises moving the handheld device in a gesturedirection that differs from a visual attribute direction used to controlthe displayed visual attribute.
 19. The method of claim 17, wherein thetap is generated by the human by one of tapping a holding hand digitonto the handheld device, tapping an opposing hand onto the handhelddevice, tapping an opposing hand digit onto the handheld device, tappinga human body part onto the handheld device, and tapping the handhelddevice onto a solid object.
 20. The method of claim 17, wherein anadditional action by one or both of the device processor and theremotely connected processor, based at least in part on one or both ofan additional selected object and an additional indication time ofdetermining the additional selected object by the human, is performedby: acquiring, by the device processor from the at least one inertialmeasurement unit, one or more of additional accelerometer data,additional gyroscope data, additional gravitational orientation data andadditional magnetic orientation data; displaying an additional displayedvisual attribute on the at least one device display that is dependent onone or more of the additional accelerometer data, the additionalgyroscope data, the additional gravitational orientation data and theadditional magnetic orientation data; acquiring, by the device processorfrom the at least one inertial measurement unit, one or more ofselection accelerometer data, selection gyroscope data, selectiongravitational orientation data and selection magnetic orientation data;and determining by the device processor from one or more of theselection accelerometer data, the selection gyroscope data, theselection gravitational orientation data and the selection magneticorientation data, one of an additional tap on the handheld device by thehuman and an additional gesture movement of the handheld device by thehuman, indicating that the displayed visual attribute is perceived bythe human to associate with the additional selected object of the two ormore perceived objects.
 21. A method for a human to select an actionusing a handheld device related to two or more perceived objects inproximity to the human, each of the two or more perceived objects havingat least one distinctive visual attribute, the method comprising:gripping, by the human, a handheld device that includes a deviceprocessor, at least one device display operatively coupled to the deviceprocessor, at least one inertial measurement unit operatively coupled tothe device processor, and at least one selection sensor operativelycoupled to the device processor; manipulating, by the human, thehandheld device, whereupon the device processor: acquires movement datafrom the at least one inertial measurement unit related to movement ofthe handheld device; and displays one of a plurality of displayed visualattributes on the at least one device display that is dependent on themovement data; activating the at least one selection sensor, by thehuman, whereupon the device processor identifies an indication by thehuman that the displayed visual attribute currently displaced on the atleast one device display is perceived by the human as being associatedwith a selected object of the two or more perceived objects; andwhereupon the action is performed, by one or both of the deviceprocessor and a remotely connected processor, based at least in part onone or both of the selected object and an indication time of theindication by the human.
 22. The method of claim 21, wherein the two ormore perceived objects each comprises one of a printed object within abook or a drawn object within a magazine.